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Introduction 



History of the project 

In 1988, researchers in the field began to focus concern on plans for evaluation of 
matriculation, both at the local and statewide level. During the 1989 Annual Asilomar 
Research Conference sponsored by CACC, NORCAL, and SCCCIRA, Jon Kangas 
suggested a strategy for promoting local research and maximizing its benefits. He 
proposed that as researchers in the field, we should develop a few simple models for local 
evaluation of matriculation, that we should publish the models and encourage their use 
wherever appropriate in the state, and that we should compile the results from use of each 
model in an aggregated report. 

The Chancellor's Office wanted to promote local research efforts in evaluating matriculation 
and was interested in providing technical assistance. Later in the spring of 1989, the 
Chancellor's Office called together a group of representatives from CACC, NORCAL, 
SCCCIRA, and the Matriculation Regional Advisory Committee. The group was charged 
with executing the plan presented at Asilomar by Jon Kangas. This document is the initial 
work product of the group, and we plan to follow-up with monographs summarizing the 
findings from colleges' use of each of the models. 



Limitations of the project 

Simplicity . A paramount concern in the development of the designs that follow was that 
they be simple. We wanted straightforward studies that were simple to perform, and 
•esults that were easily understood by all audiences. We sought designs that could be 
executed even at colleges with v^ery few resources for research. Nevertheless, to a degree, 
the strengths of this simplicity a;e balanced by methodological weaknesses. We 
acknowledge that the methodology of some of the studies could be improved through more 
elaborate design or analysis, but such changes would run counter to our goais for the 
project. 

Comprehe* ~ iveness . We have not developed a comprehensive or exhaustive evaluation of 
matriculation. Instead, we have developed a set of tools that are easy to use and fit many 
local research needs. Our purpose is to promote local research and to aggregate it wherever 
sensible, but a complete evaluation of matriculation will need to go beyond these models. 

Colleges should not assume that completing a few or even all of the designs will constitute 
a comprehensive and exhaustive evaluation of matriculation. Additional research will be 
needed, including validation studies of local placement instruments and placement 
procedures, counts of students by exemption category, and other data as required by the 
Chancellor. 

Legal constraints . Hie methodology of the designs that follow was constrained by legal 
advice from the Chancellor's Office. We were prohibited from presenting experimental 
designs that involved an untreated (or waiting-list) control group and random assignment of 
subjects to treatment conditions. As a consequence, many of the designs rely on somewhat 
more complex, yet less powerful quasi-experimental approaches. 
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The Designs 



Models . Each of the designs or studies is presented as a research template or model, and 
some of them may be adapted for use in your local district. Although you should not 
change basic features of the design without consulting with the Technical Assistant listed 
with the study, you may freely adapt the study to varying subject matter. For example, 
Study 6 might be adapted to study the relationship between reading level and political 
science performance. Also, the student satisfaction survey associated with Study 9 will 
need to be adapted for your local needs. 

Strategic s uggestion. Since many of the designs have student demographic information 
(e.g., age, gender, ethnicity, educational goal, etc.) as variables, it would be advantageous 
to develop a single common data base. This data base could be utilized with the various 
studies as certain outcome information is compiled. One of the critical variables will be the 
identification of the "served" and "not-served" matriculants. Attachment 1 in Design 9 
provides a survey form that could be used to gather the demographic data referred to in 
many of the designs. 

Technical assistance . We encourage you to contact the Technical Assistants for any of the 
designs you adopt. The Assistant is available to discuss the study and your plans, and to 
furnish technical advkx on design, execution, data analysis and reporting. 

Statistical tests . Tests of statistical significance are not required in any of the studies, 
although they are recommended if sample sizes fall below 100 per group, and because 
several of the designs do not meet the assumptions for tests of significance. We encourage 
you to focus your attention on the practical and educational significance of any differences 
you observe between groups. If the sample sizes exceed 100 per group, differences that 
are large enough to have practical importance will typically be statistically significant. 

Reporting . As part of accountability, we understand that the Chancellor's Office will be 
asking you to make available these final reports for the matriculation site review process. 
However, because our work group plans to prepare a monograph summarizing the findings 
across the colleges that perform each study, we would like you to submit your findings in 
report form to the Technical Assistant for each design. We need your help in this endeavor 
so that we can combine the results of your individual efforts to provide a broad picture. 
Consistency in reporting format for the studies is extremely important. Please note that the 
monograph will present aggregated results and will not identify individual colleges. 

We suggest that you prepare a report of your work formatted to match the model shown for 
that design. For many of the designs, we have included sample data tables to exemplify the 
reporting format. A complete report will include the following sections: 

Introduction 

Background 

Research Question(s) 
Method 

Design 

Subjects 

Procedure 
Results 

(Present data tables) 
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Discussion 

Interpretation of key results 
Limitations of the study 
Implications for practice 
Suggestions for future research 



Appendices . We have included several appendices to assist in the conduct of the studies. 
Note that the operational definitions are proposed purely for purposes of the current 
studies; they may differ from definitions used in other contexts. Also, we encourage you 
to follow the sampling guidelines if possible. 
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DESIGN 1 



Comparing the Academic Performance of Students 
who are Fully, Partially, or Not Served 
by Matriculation Services: A Retrospective Approach 

This study attempts to answer the following research question: What differences exist in academic 
performance among students who were fully-served, partially-served, or not-served by 
matriculation? The purpose of the study is to determine whether receiving varying matriculation 
services is related to measures of student academic performance. 

When comparing differences between groups of students on an outcome measure (e.g., learning 
performance), the best procedure is to follow the guidelines of an experiment. You would 
randomly select a sample from the school's population of students. Within this sample, you would 
then randomly assign its members to one of the designated groupings who are to receive some 
treatment while others receive none or a placebo condition (i.e., e?,perimental, control). However, 
the assignment to a control condition is not allowed under matriculation guideline- An alternative 
procedure, called a quasi-experimental design, is to allow self-selection into groups. For example, 
you may not be able to deny any students access to matriculation services (a control condition), but 
some students probably exist who self-selected or chose to be not-served. Thus, you may use 
them as controls. The main problem with this design, as it applies to matriculation, is that you 
usually will not know why students selected to be fully-served or not-served. Thus, if you find 
differences in outcome measures among the groups, it may not necessarily be caused by the self- 
selected treatment. In any event, the quasi-experimental design described herein is a viable 
alternative to the true experiment. 

Method 

Design 

This research design is retrospective, that is, all data are already available. You examine earlier 
records to determine which services students received and what their subsequent academic 
performances were. The actual starting time should be a period when students were able to self- 
select services. The ending time should be at least two terms later, preferably four to six. 

You will need matriculation records indicating what services, if any, were received for each 
student. Permanent student grade records must also be available. Manual searches may be done, 
but if all such data are on computer file and li iked, much time will be saved over manual efforts. 

Subjects 

Select an earlier academic term when matriculation services were reasonably in place 
(i.e., admissions, orientation, assessment, and counseling/advisement), and also a time when 
students were not required to receive rM services although they would normally conceptualized 
today as "non-exempt." 

Ideally, the student population from which to select your random sample would be all "non- 
exempt" freshmen students without prior college experience entering y^ur college during a given 
academic term. Data Processing records, special rosters, registration dates, or transcripts can be 
used to help delineate these freshmen from all other students. Should freshmen not be the driving 
force behind your ADA, select another student group provided that they were also able to self- 
select matriculation services. 
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The necessary sample size will vary depending upon the size of the "non-exempf freshmen 
population. (See the Sampling Guidelines in Appendix B.) 

Procure a listing of all "non-exempt" freshmen students and randomly select your sample using a 
table of random numbers (See Appendix B.) A shorter method (but not quite as good) is to select 
every Nth name until the necessary sample size has been reached (e.g., population=2000; 
sample=322; select every 6th name from the defined population). 

Operational Definitions of Groups 

The fully -served group consists of "non-exempt" students who (1) were admitted, (2) received 
orientation, (3) were assessed before registration, (4) received at least one counseling session 
before registration, and (5) registered. 

Ideally, the definition for the fully served group would also include the filing of an educational plan 
and enrolling only in courses designated on that plan. Should you be able to track plans and 
courses enrolled (during the first term), include these elements in your definition. Such a modified 
definition will affect those definitions which follow: 

The partially -served group consists of students who (1) were admitted, and received at least one 
matriculation service (2 through 4) but not all such services, and (5) registered. 

The not-served group consists of students who (1) were admitted and who received no 
matriculation services (2 through 4) but did (5) register, 

Procedure 

Having selected the entire sample, consult matriculation and permanent student records to 
determine into u hich of the three classification groups each student falls (fully, partially, or not- 
served). It would be ideal to have close to equal numbers of students in all three groups but this is 
unlikely. Try to obtain at least 50 students in the smallest group. Should this not happen with the 
original sample selection, randomly select more cases until the desired number is reached in the 
smallest group. 



Results 

Outcome measures and the analysis of the data 

Individual academic performances are to be obtained from transcript files. Prepare a data 
worksheet for easy recording. M^ny student outcomes may be examined but the essential ones for 
this study are as follows: 

1 . Grade-point Average - First Term. For each student, record the GPA. To summarize 
the findings, compute the mean for each group (fully, partially, not-served). 

2. Cumulative Grade-point Average. For each student, record the cumulative GPA over 
the time period selected for this study (e.g., 2-6 terms). To summarize the findings, 
compute the mean for each group. 

3 . Successful Rates - First Term. To summarize the findings, compute the mean "success 
rate" during the first term for each group (see Operational Definitions in Appendix A). 
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4 . Cumulative Successful Units. For each student, count the successful cumulative units. 
To summarize the findings, compute the mean for each group. 

5 . Retention Rate - First Term. For each group, calculate the retention rate 
(See Appendix A). 

6. Cumulative Retention Rate. For each group, calculate the cumulative retention rate. 

7 . Persistence Rate - First and Subsequent Terms. Record for each student his/her 
persistence (see Appendix A). Do the same for subsequent terms covered during the 
time period of this study. Calculate the persistence rates for each group. Thus, you will 
have percentages for each group who completed just one term, two terms, etc. 

Sample data table 

[ This tabic exemplifies the reporting formal with fabricated data.] 

Table 1 

Academic Outcomes of Fully, Partially, and Not-served Matriculants 
Entering Mountain Community College in Fall 1986 



Fully-Served Partially-Served Not-Served 
(N=110) (N = 85) (N = 50) 



OUTCOME 


Mean 


Mean 


Mean 


GPA - First Term 


2.45 


2.14 


2.03 


GPA - Cumulative 


2.63 


2.27 


2.28 


Successful Rate - First Term 


83.6% 


70.2% 


66.7% 


Successful Units - Cumulative 


38.4 


27.6 


28.9 


Retention Rate - First Term 


80.0% 


71.8% 


70.4% 


Retention Rate - Cumulative 


76.4% 


65.5% 


63.9% 


Persistence Rate 








Term 1 + 2 


84.6% 


83.6% 


78.3% 


Term 1+2 + 3 


78.3% 


71.4% 


64.3% 


Term 1+2 + 3 + 4 


69.7% 


62.3% 


48.2% 



Discussion 

Suggestions for points to cover in the Discussion Section 

1 . Interpretation of the findings. The primary purpose of this research model is to determine 
whether the level of matriculation services received is related to student academic performance. It 
is implied that fully-served students should outperform those receiving fewer or no services. 
Directly address this point for each of the outcome measures. If no differences between groups 
are observed (or even a reversal), indicate what you believe might have been the reasons. 

2. Limitations of the study. A5 pointed out in the introduction, self-selection into groups has a 
shortcoming in that you are not likely to determine why students selected as they did. This will 
limit generalizing about treatment effects. Describe in this section any departure from this design 
and mention difficulties in execution and analysis as well. 
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3. Suggestions for further research. Students who self-select certain matriculation options may be 
different in terms of ability and motivation. You may wish to mention a future study along the 
lines of contacting ? sub-sample of each group and discussing why they selected certain services. 
You may choose to use a mail-out questionnaire for this purpose. 

Research Options 

1 . If entry assessment scores are available for the entire sample, determine if basic skills (mean 
scores) are approximately the same for the three groups. 

2. If you are not familiar with statistics, and you wish lo do a refined statistical analysis of group 
differences, consult a researcher or a statistics instructor before you begin the study. A one-way 
analysis of variance may be performed on the following variables: GPA (first term as well as 
cumulative), successful cumulative units, success rates, and retention rates (fust term and 
cumulative). Differences in persistence among the groups may be analyzed using chi-square. 

3. Retain the entire data base for long-term analysis or as comparison data for subsequent studies. 

Technical Assistance 

Contact person for this design: 

Dr. Richard A. Rasor 
Research Coordinator 
American River College 
4700 College Oak Drive 
Sacramento, CA 95841 

Office - (916) 484-8166 or 484-8427 
Home -(916) 273-6449 

Available technical assistance includes consultation on design, execution, data analysis, and reporting. 



1.4 



DESIGN 2 



An Evaluation of Student Academic Success 
Before and After Implementing Matriculation 

The purpose of this study is to determine if matriculation is likely to have had any impact upon the 
academic achievement of students. More specifically, the research question is: Have matriculated 
students earned more A, B, C, or Credit grades than similar students earned prior to implementing 
matriculation services? 

The impact of matriculation upon student academic outcomes may be examined by conducting a 
retrospective study of a time period when students self-selected their services. Hence, it is 
possible to compare academic performances of fully-served students against those who received 
fewer or no services. This approach has also been called a quasi-experiment and is described 
elsewhere in Design 1. 

Another procedure compares student performances pre- and post-matriculation implementation (a 
before-after design). This approach is described below. 

The before-after model compares data from students before matriculation services were in place 
with another set of data from different studtnts (with similar characteristics) after matriculatior was 
required. Hence, your job is one of gathering current and historical student academic data, all of 
which already exist. While this design is relatively simple if data are easily examined, it has one 
serious limitation. If there is a positive change in student outcomes (e.g., post-implementation is 
better than pre-matric^ation), you will not be able to claim that matriculation services alone caused 
the change. The students from the pre-matriculation years may be different in important ways and 
other aspects of the community college system may have changed not jusi the implementation of 
matriculation. The upshot is that any claims you make must be stated with conservative 
terminology iike, "It would appear that matriculation may be related to changes/" 

Method 

Design 

You will need institutional summary counts of student grades from previous academic years. You 
then convert the sum of the different grade notations into a percentage of all grade notations for a 
given term. Such data should be available in end-of-term reports from data processing. 

It is important to compare similar groups in a before-after design. For example, if only new 
freshmen have recently been served by matriculation, you would want to compare the grades of 
that group with grades from other freshmen prior to matriculation. The idea is to compare "appies 
with apples." Try to capture 4 years of grade data before matriculation was implemented at your 
college. This can then be compared with 1-4 years of post-implementation grade data. 

Subjects 

Select the student population (or random samples of 200-300 each, see Appendix B) from different 
years prior to matriculation. Do the same for the post-implementation years. These cohorts should 
be similar in age a.id academic experience. It is also necessary to be accurate about the time when 
matriculation oegan. If such services were phased in, you will have three time periods, pre-, 
pfiase-in, and post-implementation, 
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Procedure 



Collapse data from the academic terms into academic years. Further coliapse the successful grade 
notations against those with nonsuccessful notations (see Appendix A). 

Results 

Outcome measures and analysis of the data 

Take the yearly counts of grades and convert them into percentages as shown below with the 
fabricated data: 



Grade 


Count 






A 


2,685 






B 


3,270 






C+CR 


5370 


11.325 = 


= 66% 


D 


1,849 






F 


647 






NC 


578 






W 


2,540 






I 


229 


5,943 = 


= 34% 


Total 


17,268 




= 100% 



Thus, with these bogus data, 66% of the grade notions were A, B, C, Credit, or "successful. 
Note that the percentage is based upon total grade notations and not upon student headcounts. 

Now calculate the same for several years and either graph or table the results as shown below: 

Sample data table 

I This tabic exemplifies ihc reporting format with fabricated data.] 

Table 1 

Success Rates Over Time 



Matriculation 



Year 


Implementation 


Success Rate 


1983 


Pre 


62% 


1984 


Pre 


61% 


1985 


Pre 


64% 


1986 


Phase-in 


66% 


1987 


Phase-in 


68% 


1988 


Phase-in 


72% 


1989 


Post 


73% 


1990 


Post 


73% 



Discussion 

Suggestions for points to cover in the Discussion section 

1. Interpretation of the findings. Based upon your percentage values, make a determination if 
there were noticeable changes from pre- to post-implementation periods. Also, note other 
important events like economic changes, shifting student demography, or increased rigor in 
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courses due to Title 5 changes. Use conservative language in your report as previously noted. 
If no changes were observed, indicate some reasons why. 

2. Limitations of the study. As indicated previously, the before-after design does not isolate the 
efforts of matriculation from all other variables. Should there be changes from pre to post 
periods, it may be a function of other variables, not necessarily matriculation. The before-after 
approach can be improved substantially by adding a control group 010 matriculation services 
received during pre and post periods), but such controls are not allowed under matriculation 
guidelines. Remember to make conservative claims. 

3 . Suggestions for further research. In your report you may want to include such suggestions as 
they become apparent after having done this study. Research should answer the question 
posed, but it will likely raise several other unanswered questions. 

Research Options 

1 . If computer resources are available, you can further refine the successful grade notations to 
include only those coming from selected cohorts ( e.g., gender or ethnic groups). 

2. Select certain courses and compare the pre- and post-matricuhtion successful grade rates. 



Technical Assistance 

Contact person for this design: 

Dr. Richard A. Rasor 
Research Coordinator 
American River College 
4700 College Oak Drive 
Sacramento, CA 95841 

Office: (916) 484-8 166 or 484-8427 
Home: (916) 273-6449 

Available technical assistance includes consultation on design, execution, data analysis, and 
reporting. 
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An Experimental Comparison 
of Alternative Matriculation Treatments 

This design is aimed at the following research question: Is one matriculation method more 
effective than another? The purpose of the study is to use ihe traditional experimental methods of 
the behavioral sciences to evaluate and compare the effectiveness of alternative matriculation 
treatments. The study can be used to compare two completely different matriculation programs, 
alternatively, it can compare different versions of one or more specific components (e.g. new 
orientation vs. old orientation, group vs. individual counseling). 

Questions about cause and effect are best answered by experimental studies involving random 
assignment of subjects to alternative treatment conditions, and that is the approach described in this 
proposal. Indeed, random assignment to treatments is a key assumption of the inferential statistics 
generally used to evaluate experimental findings in the behavioral sciences. 

The ethical concerns sometimes associated with random assignment *o an untreated control group 
are not relevant to this design proposal. Here, no students are denied matriculation services; 
instead, two promising alternatives are compared, with the aim of discovering which is more 
powerful. 

Because it uses random assignment of students to treatments, this design cannot be conducted as a 
retrospective study using existing data. Accordingly, plans for treatment of students and gathering 
of data must be developed before matriculants are served. 

For most institutions, the bulk of the required data for each student will be available from college 
rt n Processing services. 

Method 

Design 

The experiment will follow a between -groups experimental design with students assigned 
randomly to either a matriculation method A or B. Outcome measures will include first term 
success rates and course retention rates together with second-term success rates and persistence 
for each student. 

Subjects 

At least 200 students will participate in the study. (Guidelines for sampling are provided in 
Appendix B.) These students will be drawn from the general population of non-exempt students. 
At some colleges, a larger sample would permit breakdown of results by ethnicity. 

Procedure 

The students who will participate in the study must be identified in advance of the delivery of 
matriculation services (other than the application component). 

Each student in the study sample will be assigned randomly to receive matriculation method A or 
B. These methods might differ in a single component (e.g., group counseling vs. individual 
counseling, new vs. old orientation) or they might involve differences throughout the entire 
program. Each participating student should receive a note of explanation that might be worded as 
follows: 
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Wc arc evaluating our procedures for new students. A< p.irt of this evaluation, different students may 
complete somewhat different procedures. For yuu, [orientation (or) counseling] will involve... [describe 
as appropriate]. 

Demographic and outcome data for each student will be collected at a time and in a manner mobt 
convenient for the participating college. 

Results 

Description of the sample 

For each student, gender, ethnicity, and other salient descriptive data will be recorded. 
O'itcome measures 

For each student, the. following measures will be computed: 

First-term success rate 

First-term course retention rate 

oecond-term success rate 

Second-term persistence rate 
Each of these measures is defined in the attachment Operational Definitions in Appendix A. 

Analysis of ihe data 

The first-term mean success rate and mean retention rate will be computed for each group. The 
mean second-term success rate will be computed for each group. FLially, the frequency and 
percentages of students in each group who drop out or persist to the second term will be computed. 

Differences in group -neaps and frequencies will provide the most important findings to inteipret. 
Are these differences large enough to be educationally important? The prac tical significance of the 
group differences is more important than their statistical significance. Indeed, as long as the 
sampling yields r it least 100 subjects per group and the differences in group means and frequencies 
are large enough to be educationally important, tests of statistical significance may be considered 
optional. Tests of significance and other optional statistics are discussed briefly in the Study 
Options section. 

In addition, the gender, ethnicity, and other descriptive characteristics of the entire sample will be 
represented in frequency distributions. These data will permit comparison of the characteristics of 
the study sample to the known characteristics of the general student population. 

Sample data tables 

(These tables exemplify the reporting format with fabricated data] 

Table 1 

Mean First-Term Course Success Rates 

Method A Group = 67% 
Method B Group = 64% 



Table 2 

Wean First-Term Retention Rate: 

Method A Group = 65% 
Method B Group = 69% 
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Table 3 

'Mean Second-Term Course Success Rates 



Method A Group = 72% 
Method B Group = 75% 



Table 4 

Observed Frequencies of 
Persistence to Secotid Term by Group 



Oatconvg 
Persisted Dropped 



Method 



B 





57 (38%) 


72 (48%) 


78 (52%) 



Discussion 

Suggestions for points to cover in tke Discussion section 

1 . Interpretation oftfit findings. What do results mean? It is especially important to discuss 
the practical and educational importance of any observed differences between groups. Are the 
differences in effects of the methods great enough to justify the differences in resource 
expenditures? 

Results like the fabricated data shown . " ove might indicate that Methods A and B produce 
roughly equal course success and retention, yet Method A yields greater persistence to the 
second term. 

2. Limitations of the study. If the study sample was not typical of the general population, 
describe the differences and caution the reader to limit generalization of the findings to similar 
students. Describe any departure from the design proposed herein. Mention any difficulties 
that occurred in execution of the study and describe how these may have affected the findings. 

3. Suggestions for further research. In view of the results, what additional research seems to be 
needed? One likely suggestion would be choice of a two-factor design, crossing treatment 
with ethnicity in order to determine whether the differences between methods A and B are 
similar for all groups. 

Study Options 

1 . Tests of statistical significance may be useful if the observed differences between groups are 
small, or if sample sizes fall below 100 per group. Differences in group means may be tested 
with one-tailed r-tests using the .05 level of significance and the differences from expected 
frequencies will be tested with chi-square at tiie .05 level of significance. Computational 
procedures for Mests and chi-square are described in any introductory college-level statistics 
text. 

Even if tests of significance are not performed, differences in group means may be interpreted 
more easily if the standard deviation is computed for the success and retention rates of each 
group. The differences between group means that are attributed to matriculation effects can 
then be compared to the natural variation in student performance found within each group. 



2. An option that would enhance the information value of the proposed study is to perform a 
two-factor design, crossing treatment with ethnicity. With this approach, there would be a 
method A and method B group for each ethnic category, with students assigned randomly to 
these "cells". Each ceil should contain at least 30 students and tests of statistical significance 
will be required unless all cell frequencies exceed 100. Two-way analysis of variance may be 
used, reporting cell means, as well as F-ratios, degrees of freedom and probability levels for 
the effect of method, ethnicity, and the method by ethnicity interaction. 

3. Another variation involves use of existing data for a retrospective comparison of alternative 
matriculation programs or components. For example, your college might have changed its 
orientation program recently and might have the necessary student outcome data on file for 
samples of students from before and after the change. 

Because of its relative ease, this sort of study appears attractive, but compared with the power 
of an experimental design involving random assignments to treatments, the power of this 
design is weak. With this sort of retrospective study, no causal attributions can be made. If 
students who completed the new orientation perform better, it is not clear whether that 
difference is attributable to the new orientation or to other changes that accompanied its 
introduction (e.g. increased staff enthusiasm, increased funding for matriculation that lead to 
generally improved services, changes in enrollment patterns, etc.). Further, traditional tests of 
statistical significance are inappropriate because the assumption of random assignment is 
violated in this design. This design is a weak second choice that should be used only if 
random assignment to treatments is impossible, even for a limited sample of students. 



Technical Assistance 

Contact person for this design: 

Dr. William E. Threlfall 
Coordinator of Institutional Research 
South County Community College District 
25555 Hesperian Blvd. 
Hayward, CA 94545 

(415) 786-6698 



Available technical assistance includes consultation on design, execution, data analysis, and 
reporting. 



DESIGN 4 

Disproportionate Impact and Writing Assessment 



This design is aimed at the following research questions: 1) Does a disproportionate impact occur between 
testing and entrance into remedial courses? 2) Does a disproportionate impact occur between remedial 
coursework and subsequent college-level coursework The purpose of the study is to organize testing, 
placement, and enrollment data in such a way that it :s possible to see the impact on ethnic groups as they 
go through the testing, placement, and enrollment process. 

This is a retrospective study for colleges that have access to test scores broken down by ethnicity and for 
colleges that are able to distinguish between AA-applicable English and ESL writing courses and NonAA- 
applicable English and ESL writing courses. This study only addresses the ethnic variable and writing 
assessment. However, the model can be used to do an analysis of the impact of other assessments (e.g., 
reading) in relationship to other variables with civil rights implications, such as sex or disability. 

Both Federal Civil Rights Laws and the Title 5 Assessment Regulations ask us to address issues of 
disproportionate impact. If an ethnic group is represented in basic skills courses in a significant^ different 
proportion than in the tested group, we are asked to be sure that the test is a valid one. Data are included in 
this study indicating the proportions of ethnic groups qualifying for remedial courses to further clarify 
what is happening in this process. 

If an ethnic group is represented in college-level courses after basic skills coursework in a smaller 
proportion than other ethnic groups, a disproportionate impact is said to occur and colleges are asked to 
explain why this has occurred and to develop and implement appropriate intervention strategies to correct 
the situation. 

Method 

Subjects 

Tested population (Table 1, Column 1). Students who were tested for writing placement in either English 
or ESL between May 1, 1988 and October 1, 1988, and who first enrolled at this college in Fall 1988 
should be included in the study ind will be referred to as the 'Tested" population. The ethnic composition 
of this group should then be determined. Ethnic categories are defined in Appendix A, however, each 
college may use an expanded version for local needs. 

NonAA-placed (Table 1, Column 2). The "Tested" population should be broken down into two groups: 
those who qualified for nonremedial writing and those who qualified for remedial writing (including ESL 
writing) based on the results of the placement test. The remedial group should then be broken down into 
its ethnic groupings and will be referred to as the "NonAA-Placed" group. 

NonAA-e.nr oiled population (Table 1, Column 3). Next determine who, of the Tested population, enrolled 
in NonAA-applicable English or ESL writing courses for Fall 1988. Then break this group down into its 
ethnic groupings. This group will be referred to as the "NonAA-enrolled" population. 

NonAA-nersisters (Table 1, Column 4): Next, determine which of the NonAA-enrolled population 
persisted to the next term and took one or more AA-applicable courses. Then break this group down into 
its ethnic groupings. This group should be referred to as "NonAA-persisters." Persistence is said to exist 
if the student had any grade on his/her subsequent term transcript (see Definitions in Appendix A). 
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Results 



Sample data tables 

(These tables exemplify the reporting format with fabricated data.] 

The following table indicates the number and percent of students b\ ethnicity in the Tested, NonAA- 
placed, NonAA-enrolled, and NonAA-persisting groups. 

Table 1 

Number and Percent by Ethnicity of Tested, NonAA-placed, 
NonAA-enroiled and NonAA-persisters 



Ethnic Group N 



Group 1 
Group 2 
Group 3 
Group 4 



Col. 1 


Col. 2 


Col. 3 


Col. 4 


Tested 


NonAA- 


NonAA- 


NonAA- 




Placed 


Enrolled 


Persisters 


N % 


N % 


N % 


N % 


800 20% 


500 25% 


360 30% 


320 40% 


800 20% 


500 25% 


360 30% 


160 20% 


1200 30% 


500 25% 


240 20% 


80 10% 


1200 30% 


500 25% 


240 20% 


240 30% 



Total 



4000 100% 2000 100% 1200 100% 800 100% 



Table 2 indicates the percent difference between Tested, NonAA-placed, NonAA-enrolled and NonAA- 
Persisters. 

Table 2 

Percentage Differences between Tested, NonAA-Placed, 
NonAA-Enrolled, and NonAA-Persisters 

% Diff. % Diff. %Diff. 
Ethnic Group Col. 1&2* Col. 2&3 Col. 3&4 

Group 1 +25% +20% +33% 

Group 2 +25% +20% -33% 

Group 3 -17% -20% -50% 

Group 4 -17% -20% +50% 

*Column numbers refer to those used in Table 1 above. An example calculation using Group 1, Column 1 and 2 
is as follows: Column 2 (25%) - Column 1 (20%) = +5% divided by Column 1 (20) = +25%. 

Comments 

1 . Comment on percentage differences in ethnic composition of Tested and NonAA placed 
groups. 

2. Comment on percentage differences between the NonAA-placed and NonAA-enrolled groups. 

3. Comment on percentage differences between the NonAA-enrolled and NonAA-persisters. 



0 
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Discussion 



Suggestions for points to cover in the Discussion section 

1. Interpretation of the findings. Differences between Tested and NonAA-placed populations. Are the 
proportions different between these two groups? Why is there a difference? What factors may relate 
to a particular ethnic group hav ing a higher or lower proportion that qualify for remedial courses? Are 
there any questions of test validity that need to be raised? 

Differences between No nAA-placed and NonAA -enrolled populations. Do the various ethnic groups 
enroll in remedial courses in different proportions than they qualified for remedial courses? What 
factors may relate to differences if they exist? Are there limitations in certain sections of remedial 
courses that may impact one ethnic group more than another? Do different ethnic groups go through 
different assessment processes that may result in their being more apt to enroll into remedial courses 
than other ethnic groups? 

Differences between NonAA-enrolled and NonAA-persisters. . Did the group of NonAA-enrolled 
students persist to the next term in different proportions than they existed in the NonAA-enroIled 
group? If differences exist, why might they be there? 



2. Suggestions for future research. Further research typically will be needed to determine why any 
particular difference exists. You might suggest that analyzing differences based on the original level of 
writing may shed some light. You might want to suggest calculating the proportion of remedial to non 
remedial units taken by the NonAA-enrolled and the NonAA-persisters to see if the Non AA-en rolled 
students are in fact taking a greater proportion of AA-applicable courses over time. 



Technical Assistance 

Contact person for this design: 

Dr. Jon Alan Kangas 

District Dean of Academic Standards 

San Jose/Evergreen Community College District 

4750 San Felipe Road 

San Jose, CA 95135-1599 

(408) 270-6466 

Available technical assistance includes consultation on design, execution, data analysis, and reporting. 
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DESIGN 5 



Evaluating the Effectiveness of Remedial Placement 



The Matriculation Bill, AB 3, requires evaluation of the success rate of students who enroll into 
recommended courses compared to students who choose not to follow colleges' advice. The 
purpose of this study is to compare the performance and persistence of students who enrolled into 
and completed the recommended remedial reading course with students who did not enroll into the 
suggested course. The research question addressed is: Does successful completion of 
recommended remedial reading coursework assist students in their performance and persistence in 
college? 

This study can be conducted retrospectively utilizing existing college data. Students' initial 
placement information will need to be available in addition to student historical transcript records. 
This design is based on the supposition that students who completed the assessment component 
and received information on their placement were apprised of the college's recommendations 
regarding remediation either through a group interpretation of placement results during orientation 
or a counseling session. This design could be used in evaluating the effectiveness of any remedial 
or developmental course, provided recommendations regarding remedial class enrollment were 
made to students. 

Method 

Design 

The study will consist of two groups: the advice followers and the non-followers of advice. The 
measures of student success will include the percent of units completed satisfactorily during their 
first term and students' persistence and performance in the following term. 

Subjects 

The students will be those assessed and placed into the highest level of remedial reading. The 
sample to be studied should include only those students v/ho participated in th? college's 
matriculation activities (i.e., admissions, assessment, orientation and counseling/advisement) 
during a set period (e.g., Fall 1988). A minimum of 200 students should be included in the 
study. The sample should include nearly the same number of students placed into the specific level 
of remedial reading who enrolled in the course and students who were placed into the same level of 
reading but did not enroll in the reading course. Sampling techniques could be utilized 
(see Sampling Guidelines in Appendix B). To determine the population size, it will be necessary 
first to identify the number of students assessed and placed into the highest level of remedial 
reading who also participated in the matriculation program. 

Procedure 

This study is best undertaken by schools who can identify their "served" matriculants and can 
easily match placement results to students. It would be advantageous for transcripts to be "on- 
line," however, it is not necessary. If the number of students in each group could be expanded, it 
would allow for a more detailed breakdown of results to include an examination of the interaction 
of ethnicity, age, gender, and educational goal upon "advice following behavior" regarding 
remediation and subsequent student performance and persistence. 
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Results 



Outcome measures 

Each student's success rate (see Operational Definitions in Appendix A) will be calculated for all 
courses during his/her first term. Additionally, the following term persistence (see Definitions) for 
each student will be noted. The student's success rate for the following term will also be 
determined. Students who enroll in the recommended remedial reading class during their second 
term should be identified as such. 

Analysis of the data 

The Advice Followers group should be further divided into those who successfully completed 
(CR,"C"or better) or one or more units of the remedial reading course and those who did not. The 
mean success rate should be calculated ior each of the three groups. (Advice Followers-Course 
Completers and Course Non Completers „nd Non-advice followers.) The percentage of each 
group who persisted to the next term should be calculated (Persistence Rate - see Definitions). 
After the conclusion of the second term, the mean success rate should be calculated for each of the 
groups. 

Description of the sample 

If sampling techniques were utilized, gender, ethnicity, age, and other descriptive characteristics of 
the entire sample should be represented in frequency distributions. These data will permit 
comparisons of the study sample to the known characteristics of ihe general student population. 

Sample data table 

[This tabic exemplifies the reporting format with fabricated data.) 



Table 1 

Success and Persistence of Remedial 
Advice Followers and Non-Advice Followers 



OUTCOMES 



GROUPS 



Number of 
Students 



1st Term 
Mean 
Success 
Rate 



Mean 

Persistence 
Rate to 

Following Term 



Following 
Term - Mean 
Success Rate 



Remedial Students- 
Advice Followers: 
Course Completers 
Course Noncompleters 



180 
85 



72% 
53% 



78% (140) 
49% (42) 



78% 
54% 



Remedial Students- 
Non-advice Followers 



237 



66% 



51% (120) 



63% 
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Suggestions for points to cover in the Discussion section 

1 . Interpretations of tfie findings. What relationship does remediation have with increasing 
students' ability to succeed in their coursework and persist in the pursuit of their educational 
and vocational goals? The remedial classes in which the students were placed are being 
scrutinized for their effectiveness in better preparing students for college-level coursework. 
However, since performance of students who choose to follow advice or not ein^ 
measured, there is a strong possibility that the findings may reflect other contributing 
characteristics inherent in advice followers and non-advice followers. 

2. Limitations of the study. If the assessment and placement program has not been evaluated, 
student placements may not be valid and therefore reflect a misleading need for remediation. 
Students may not take the assessment process seriously and put forth their best effort, thereby 
contributing to inaccurate placement. 

If possible, identify the students who were advised to enroll into remedial reading classes, but 
were unable to do so due to lack of availability of the class. 

If modifications to the design were incorporated during implementation of the study, please 
identify what they were. Describe any difficulties that may have arisen during the course of 
the study and what impact they may have had on the results. 

3. Suggestions for further research Upon reflection of the results and the study, what \ariations 
to this type of investigation appear to be necessary? 

Study Options 

1 . One variation that may contribute to the usefulness of the findings is to identify the success 
rate of the three groups for units that had transferable status. 

2. These three groups of students could be flagged and tracked over succeeding terms to 
determine the long-term impact of following advice and consequent successful completion of 
remedial coursework. The groups would need to be controlled for iaier completion of 
remedial coursework. The length of time required for students in the groups to complete an 
Associate Degree or transfer to a four-year institution could be calculated. 

3. Students who enroll into six (6) or more units and indicate an educational goal of earning an 
Associate Degree or transferring on to earn a Bachelor's Degree could be identified and 
analyzed separately. This would limit the ability to generalize results to the entire population; 
however, it would focus on a targeted group. The value of remediation and tendencies to 
follow advice could be examined as it affects the performance and persistence of these 
students. 

4. Include other pertinent demographic variables such as ethnicity, age, and gender to identify 
other factors contributing to students' advice-following behavior and ^alue of remediation. 
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Technical Assistance 



Contact person for this study: 

Marty Dunlap 
Assessment Coordinator 
Butte Community College 
3536 Butte Campus Drive 
Oroville,CA 95965 

(916) 895-2350, ext. 646 



Available technical assistance includes consultation on design, execution, data analysis, and 
reporting. 
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DESIGN 6 



History Course Success 
Based On English Eligibility 



The legislative intent in AB3 and its guidelines for assessment and placement speak to a 
desire to guide students in determining and understanding the skills rieeded for programs 
and coursework. In presenting the Title 5 Regulations, the September 1989 Board of 
Governors' document speaks of matriculation as "a program essential to and interrelated 
with the provision of basic skills instruction and rigorous course standards necessary to 
ensure student access and success." The goal then of combining assessment, placement, 
and counseling is to aid the student in developing a comprehensive plan to achieve a 
particular educational goal based on his or her interests, abilities and desires. 

This design will evaluate the viability of using student results from English language skills 
assessments, or English writing course completion for placement into other collegiate level 
coursework. More specifically, this effort will review student success in the introductory 
.US, History transfer course as it relates to his/her English language course eligibility 
and/or completion. 

This study is designed for colleges that: 1) have an English placement procedure based on 
student performance on standardized tests and/or college coursework; 2) have completed 
Title 5 (strengthening standards) activities in the History department; and 3) either do not 
have a prerequisite policy restricting entry into the History class, or restrict only a minimum 
number of students. 

This study may be done either retrospectively or prospectively. Because the project 
requires tu . semesters of data, it is recommended that you do this study retrospectively. It 
may be helpful io review several semesters to insure that patterns are constant. Technical 
support irom data processing staff will be helpful. It will be necessary to mati;h studer* to 
their assessment/placement levels and previously completed coursework. 



All students fitting the subject criteria should be considered. The data to be gathered 
include: 

U.S. History grades for students who entered the college in the Fall 1988 term; 
their English language placement level based on assessment; and/or their English 
writing class with successful grades (CR, U C" or better) from the Fall 1988 term 
organized by exiting placement level; and additional demographic variables: gender, 
ethnicity, and age. 



Subjects for this study would include all students enrolled in U.S. History a s of the first 
census in the Spring 1989 term. In consideration of possible writing skill development of 
students due to time lapse between skills assessment or general course enrollment and 
enrollment into the U.S. History class , only first time students new in Fall 1988 should 
included in the study. 



Method 



Design 



Subjects 



Procedure 



It is strongly recommended that assistance from the college's data processing department 
be obtained to construct the student data files and initial comparison tables. Trying to do 
the project manually would be very difficult. Due to the study limitations and because of 
the phasing in of Title 5 standards, schools may choose to review previous groups 
matching the same sample limitations, noting when curriculum modifications took place. 
Data processing should be able to produce the comparison data shown in Table 1 of the 
Results section. 

A separate data file should Ix, created for each term studied. All files can be combined if the 
distributions of the outcome measures (success rates) look very much the same from term 
to term. If not, they should be analyzed separately with attention to reasons for the 
variation. 



Outcome measures 

The student's success in U.S. History is the outcome measure, and a positive relationship 
between level of English eligibility and history success is expected. 

A table of the percentage of students with successful completion of the history course (see 
Operational Definitions in Appendix A) by each level of English will be compiled (see 
example below). Tables for each semester and subgroup populations can be developed to 
illustrate the relationship between English skill levels and history success. 

Sample data table 

[This tabic exemplifies the reporting format with fabricated data.] 



Results 



Table 1 



Percent of Students Achieving Successful US. History Grades 
by English Level based on Assessment Scores 



English Level* 



% Successful in History 



English 1A 



75% 



English 1LB 



65% 



English 2 LB 



30% 



English 3 LB 



25% 



English 4 LB 



10% 



LB" stands for level below Freshman Composition. 
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TabJe 2 

Percent of Students Achieving Successful U.S. History Grades 
by English Level based on Prior English Course Completion 



Enelish Level** 



% Successful in History 



English 1A 



85% 



Engish 1LB 



65% 



English 2LB 



50% 



English 3LB 



30% 



English 4LB 



2Q& 



** Students have successfully completed English 
courses at these levels. 



Discussion 



Suggestions for points to cover in the Discussion section 

1 . Interpretation of the findings. Does English placement or the completion of an English 
course "predict" success in the history course? The percentages of successful students 
at each level of English should provide evidence for the establishment of an appropriate 
prerequisite level for U.S. History . 

2. Limitations of the study. Even though some colleges have prerequisites for U.S. 
History in place, they may be advisory at this time or they may be quite low. In this 
case, you can undertake this project if there is a limited number of students restricted 
from enrolling into the U.S. History course. 

ESL writing courses considered to be at the same level as English courses must initially 
be considered separately. If there are multiple paths of entry into the English courses, 
such as different tests, coursework, high school transcripts, etc., additional subgroups 
for the independent variable (English level) should be considered. 

Instructor, content, and grading variations can significantly influence student 
outcomes. Different standards and different text reading levels are clearly part of the 
"treatment" we call education, yet frequently defy analysis. Participating colleges 
should strive to make their sample representative of their student population while 
endeavoring to insure that the course treatment (in both English and History 
courses) is as homogeneous as possible. 



1 . The calculation of a chi-square between the variables can verify a statistically significant 
relationship. 

2. Another basic variation of this project is to compare the actual English grades to the 
U.S. History grades. Variations in grading standards will likely be highlighted. 



Study Options 
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3 . Additional variations of this project require considerable data processing support 
and statistical analysis. A more rigorous analysis of factors available through 
utilizing student demographics data could be undertaken. 



Technical Assistance 

Contact person for this design: 

Scot L. Spicer 
Planning and Research 
Glendale Community College 
1500 N. VerdugoRoad 
Glendale, California 91208 

Office: (818) 240-1000 extension 413 
Home: (213)839-8380 
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DESIGN 7 

Student Access and the Implementation of Matriculation 



The purpose of this study is to determine the impact of required matriculation activities upon 
student access and enrollment in your community college. The issue of access may be considered 
at different points in time; from community populations to applying for admission, or from 
application to actual registration. This study will focus on access during the time period from 
application submitted to registration for classes. 

The research questions posed by this study are: 

1 . Have matriculation activities (i.e., admissions, orientation, assessment, 
counseling/advisement) made any difference in overall application-to-registration rates? 

2. Have required matriculation activities (i.e.admissions, orientation, assessment, 
counseling/advisement) created any disproportionate impact on ethnic groups in 
admissions-to-registration rates? 



Method 



Design 

This approach will examine the percentage of new, non-exempt applicants who subsequently 
registered for classes. Data must be broken down by ethnicity and by different time periods 
representing pre-, phase-in, and post-matriculation implementation. As such, the design will 
require centralized data processing assistance. Colleges participating in this design will need to 
have maintained records of applicants who did not register for classes. 

Subjects 

Several terms of all new freshmen applicants (submitted applications) who would be considered 
as "non-exempt" will comprise the population for this study. It will also be necessary to establish 
three time periods: 1) two-four academic terms before implementing matriculation; 2) phase-in 
terms (if applicable); and 3) two-four terms after required matriculation activities were firmly in 
place. All new freshman applicants for each of these terms must be identified and their ethnicity 
established. A later census count will not be appropriate. It is presumed that during the post- 
implementation terms, all such students received orientation, assessment, and at least one 
counseling session before being allowed to register for classes. 

Procedure 

The new freshman applicant group and their ethnicities (see Operational Definitions In Appendix 
A) will be identified for each term during the pre-, phase-in, and post phases of matriculation. 
The exact manner in which this is done will depend upon data processing procedures. You will 
only need summary counts of students rather than individual identities. Of course, if you wish to 
do follow-ups with nonregistrants, you will need names, identification numbers, or other means 
in order to identify these students. 
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Results 



Analysis of the data 

Prepare tables like those shown below for each term during the three phases of implementation 
(pre-, phase-in, post). 

After having put together several such tables representing the pre-, phase-in, and post- 
implementation terms, you may compute an average percent for each group representing each time 
period (average of pre-, average of phase-in, average for post-). An example of this is presented 
as Table 3. The variation in initial (pre-) percentages should help you to establish a reasonable 
baseline for each ethnic group. The question to address is: Has the rate changed since 
implementing matriculation? 

Assume that the sample data in Tables 1 and 2 now represent composite percentages as portrayed 
in Table 3. First of all, did required matriculation activities make a difference in overall 
admissions-to-registration rates? The values to compare are 99.48% (before matriculation) with 
98.83% (post-matriculation). The difference of 0.65% is small and may merely reflect routine 
changes. On the other hand, if each pre-implementation period is around 0.65% higher than each 
post-implementation period, this might suggest that required matriculation activities do affect 
registration. It would then be important to contact some of those individuals who do not register 
as a follow-up. 

Secondly, you need to determine if the required matriculation activities created any 
disproportionate impact on ethnic groups. Assuming Table 3 represents averages of more than 
one term, you will note there is very little difference except for Hispanics (99.48% pre-io 96.30% 
post). The difference of 3. 18 percentage points may very well signal a differential impact. The 
finding could be further examined by conducting short interviews, focus groups or surveys. 

Sample data tables 

[These tables exemplify the reporting format with fabricated data. Additional ethnic categories arc optional.] 



Table 1 



Application to Registration Rates for Selected Student Groups 
During Fall 1986 Semester (A Pre-Matriculation Term) 



Total 



Total 



% Applicants 
Who Registered 



Group 



Applicants 



Registered 



Asian 
Black 



4,300 
5,600 
6,358 
5,800 
442 



4,295 
5,542 
6,325 
5,786 



99.88 
98.96 
99.48 
99.76 
2L42 



Hispanic 



White 
Other 



Totals 



22,500 



22,383 



99.48 



Table 2 

Application to Registration Rates 
for Selected Student Groups During Fall 1990 Semester 
(A Post-Matriculation Implementation Term) 



Group 


Total 

Applicants 


Total 

Registered 


% Applicants 
Who Registered 


Asian 

Black 

Hispanic 

White 

Other 


4,480 
5,750 
6,480 
6,200 
642 


4,470 
5,733 
6,240 
6,193 
64Q 


99.78 
99.70 
96.30 
99.89 
99,69 


Totals 


23,552 


23,276 


98.83 


Table 3 

Composite Table Showing Averages of Applicants to 
Registrants Percentages 




Average Percents During Matriculation 




Group 


Pre- 


Phase-In 


Post- 


Asian 

Black 

Hispanic 

White 

Other 


99.88 
98.96 
99.48 
99.76 
98.42 


99.75 
99.32 
97.63 
98.42 
99.36 


99.78 
99.70 
96.30 
99.89 
99.69 


Overall 


99.48 


98.96 


98.83 



Discussion 

Suggestions for points to cover in the Discussion section 

1 . Interpretation of the findings. Small ups and downs in the percentage values probably 
represent nothing more than routine variation. Consistent directional differences would 
likely indicate otherwise. Discuss where any differences exist and include likely plans for 
finding out why this occured. 

2. Limitations of the study. Any "before-after" design using different students at different 
times without utilizing a control group allows for much interpretation should differences be 
noted. Were there other changes occurring in addition to the implementation of matriculation? 
Could any of these have affected the percentage of applicants to registrants? This type of data 
indicates what is but not why. 

3 . Suggestions for further research. Should differences exist, you will need to address the 
why with follow-up studies. Describe in this section what type of studies are anticipated. 
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Technical Assistance 



Contact person for this design: 

Dr. Richard A. Rasor 
Research Coordinator 
American River College 
4700 College Oak Drive 
Sacramento, CA 95841 

Office: (916) 484-8166 or 484-8427 
Home: (916)273-6449 

Available technical assistance includes consultation on design, execution, data analysis, and 
reporting. 
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DESIGN 8 



Monitoring Changes in Student Goal 
Before and After Matriculation 



This design is aimed at the following research question: Do students change or clarify their goal after 
participating in assessment, orientation, and counseling/advisement? 

The matriculation process was conceived to help students clarify their educational goals and to develop a 
plan to reach those goals. A more efficient use of educational resources is one desirable outcome of this 
process. Greater retention of students is also hoped to follow when students have a clear vision of their 
goal(s). 

This study can best be conducted by colleges that have their matriculation process structured in such a way 
that students can indicate their primary educational goal at the time of admissions and then again during, or 
immediately after, their counseling appointment. Since most colleges do not have thL type of student goal 
information already collected, this design cannot be conducted as a retrospective study using available 
data. 



Method 



Design 

Admissions Goal. 

The educational goals used in the MIS dictionary as SB 14 should be used during the admissions process. 
At the time of admissions, the student should indicate his or her primary goal. This goal will be referred to 
as the "admissions goal." If your college has not begun using these goals, an alternative method for 
collecting students' "admission goals" will be necessary. One suggestion for gathering these goals is to 
give each new nonexempt student, along with the usual admission materials, a separate sheet with 
admissions goals listed. The sheet should have a place for the student's name, social security number, and 
a list of the admission's goals. The student should indicate his or h^r primary goal on this sheet. 

Matriculation Gnal 

The educational goals listed in the MIS Dictionary as SMOl should be used in the counseling process and 
will be referred to as the "matriculation goals." SB14 and SMOl are identical and can be viewed as a pre- 
and post test taken before and after the matriculation process. Clarification of goals made during the 
process of matriculation should be reflected in goal changes between SB14 and SMOl. It is suggested 
that students clarify their admissions goal in the context of their counseling appointment. A sheet identical 
in content to the one used to collect the admissions goal should be used. 
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The goals to be used in the study are SB 14 and SM01 as follows: 

Obtain a bachelor's degree after completion of an associate's degree. - BA & AA 
Obtain a bachelor's degree without completing an associate's degree. - BA w/o AA 
Obtain a two-year associate's degree without transfer.- AA w/o Transfer 
Obtain a two-year vocational degree without transfer. - Voc. AA w/o Transfer 
Earn a vocational certificate without transfer.- Cert, w/o Transfer 
Discover/formulate career interests, plans, goals. - Discover Interests 
Prepare for a new career (acquire job skills). - Prepare Career 
Advance in current job/career (update job skills).-Advance Career 
Maintain certificate or license (e.g., Nursing, Real Estate). - Maintain Cert./Lic. 
Educational development (intellectual, cultural). - Educ Dev. 
Improve basic skills in English, reading or math.- Improve Bas.Sk. 
Complete credits for high school diploma or GED. - Complete H.S. units 
Undecided on goal. - Undecided 



Subjects 

In order to have sufficient numbers for each educational goal to determine meaningful changes, it would be 
preferable if all students who qualified to go through the matriculation process were asked to indicate an 
admissions goal and a matriculation goal. Only those students who did indicate both an admissions goal 
and a matriculation goal should be used in the study. 

If you cannot have all matriculated students fill out their goals, and you want to scale back the number of 
students who complete the goal sheets, ask for technical assistance. 



Results 

Analysis of the data 

The admissions goal sheets should be matched by name and social security number to the matriculation 
goal sheets. Only use data for students who have both an admissions and a matriculation goal. Use a grid 
similar to Table 1 and place a mark in the cell that corresponds to the admissions goal and matriculation 
goal for each student. If the goal is the same for both, the mark will always appear in one of the diagonal 
cells. 

Sample data tables 

[The tables exemplify the reporting format with fabricated data. ] 

Table 1 reflects the number of those students who indicated a particular admissions goal and then chose the 
same or different matriculation goal. 

Table 2 reflects the percent of students who chose a particular admissions goal and then chose the same or 
different matriculation goal. 
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Table 1 

Number of Students Who Choose an Admissions Goal Who Later Choose a Matriculation Goal 



Admissions 
Goals ba 



BA&AA 



BA w/oAA 

AA w/o 
Transfer 

Voc.AA 
w/o Trans. 

Cert, w/o 
Transfer 

Discover 
Interests 

Prepare 
Career 

Advance 
Career 

Maintain 
Cert./Lic. 

Educ. Dev. 

Improve 
Bas. Sk. 



Complete 
H.S. Units 

Undecided 

Column 
Total 



BA w/o AA w/o 
_M Transfer 



# 



# 



500 400 



VocAA 
w/o Trans 



Matriculation Goals 

Cert, w/o Discover Prepare Advance Maintain 
Transfer Interests Career Career Ccrt./Lic. 



Educ. Improve Complete Undecided Row 
-Dev Bas, ?k , H.S Units ToM 



# 



# 



# 



# 



# 



# 



# 



# 



# 
0 

0 
5 



# 



300 
400 
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Table 2 

Percent of Students Who Choose an Admissions Goal Who Later Choose a Matriculation Goal 



Admissions 
Goals da 



BA&AA 



BAw/oAA 



AAw/o 
Transfer 

Voc.AA 
w/o Trans. 

Cert, w/o 
Transfer 

Discover 
Interests 

Prepare 
Career 

Advance 
Career 

Maintain 
Ccrt./Lic. 

Educ. Dev. 

Improve 
Bas. Sk. 

Complete 
H.S. Units 

Undecided 

Column 
Total 



B A w/o AA w/o 
AA Transfer 



% 



Voc. AA 
w/o Trans. 



% 



Matriculation Goals 

Cert, w/o Discover Prepare Advance Maintain 
Transfer Interests Career Career Cert./I.ic, 



% 



Educ. Improve Complete Undecided Row 
-DSL Bas. Sk. H.S. Units ToiaJ 



% 



% 



% 



15 
20 




00 
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Comments 



1 . Comment on which admissions goals are chosen the most and least frequently. For example, one 
could note from the data that while 20% of the students want to go on to' a BA after receiving an AA 
degree, only 10% want a certificate. 

2. Comment ei which matriculation goals are chosen the most and least frequently, from the 
bogus data, one could say that while 25% of students wish to obtain an B A degree after an AA, 
15% are seeHng a vocational AA without transferring. 

3. Comment on the overall goal changes from the admissions goals to matriculation goals. 

4. Comment on significant changes from a particular admissions goal to particular matriculation 
goal(s). For example, one could note from the fabricated data that 2% of the students who 
originally had the goal of getting a BA after an A A degree changed their goal to getting a BA without 
an AA degree after the matriculation process. 

Discussion 

Suggestions for points to cover in the Discussion section 

1 . Interpretation of the findings. Discuss which goals showed the most chrnge. Discuss the overall 
rate of change. Discuss whether or not and why the matriculation process seemed to relate to change 
or clarification in student goals. 



2. Suggestions for further research. You may want to suggest further research identifying the 
demographics associated with students who change their goals. What kind of students are most 
apt to change their goals? What would happen if another study had a control group in which 
students filled out a second set of goal choices but without the matriculation process? Over time, 
how many students attain their stated matriculation goals? 



Technical Assistance 

Contact person for this design: 

Dr. Jon Alan Kangas 

District Dean of Academic Standards 

San Jose/Evergreen Community College District 

4750 San Felipe Rd. 

San Jose, CA 95135-1599 

(408) 270-6466 



Available technical assistance includes consultation on design, execution, data analysis, and 'eporting. 
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Student Satisfaction With Matriculation 

This study attempts to respond to state directives b> urveying students about their access, level of 
awareness, and satisfaction with specific campus services related to the matriculation process. 
This design may also be used to demonstrate the impact of matriculation on colleges' services. The 
ability of students to make "wise educational choices" requires the use of services and the 
application of assistance and information they receive to their decision making. 

An array of data will be generated with this survey regarding student satisfaction and use of 
campus services. If this is a first effort, we would expect that students who participated in 
matriculation and were exposed to matriculation services would show greater understanding and 
direction within the college than students who did not participate. If baseline recognition and 
satisfaction data already exist at the college, these measures may show positive increases as 
matriculation efforts are expanded to include more services and extended to more students. 

It is recommended that participating colleges have all student matriculation services 
(i.e., admissions, assessment, orientation, counseling/advisement) in place, but it is not necessary 
to have all students fully participating in these services. In fact, if matriculation services do not 
reach all eligible students for reasons of budget and/or time constraints, this allows coiicges the 
opportunity to have a naturally occurring comparison group in the research project. In this case, it 
is necessary to identify students who have received "full" matriculation services and those who 
have not. 

The ability to input data into either a personal or mainframe computer is beneficial for this project. 
The study will create a data base which can be utilized over timt for future evaluation studies. 
Availability of a statistical package on the computer is highly desirable. The capability to match 
survey information with student records through the campus mainframe is necessary only as an 
option for a different approach to the study. This study has many options, some of which would 
be helpful to integrate into surveys administered in alternate years. 



Method 

Design and development of survey instrument 

This study will examine student perceptions and satisfaction with respect to matriculation services 
through data collected by a surve> instrument. A complete survey can be developed from materials 
included witii this design or colleges may elect to develop their own survey or individual survey 
questions based on locally identified interests. The survey should reflect input from the college's 
Matriculation Committee. 

Attachment 1 provides sample survey questions to obtain basic demographic characteristics. 
Survey questionnaires where the respondent is anonymous tend to provide the researcher with 
more honest data. However, if the college has the resources to match survey responses to 
individual student records on the campus mainframe, ^eater analysis is possible. By asking for 
the student's identification number on the survey instrument, certain demographic and performance 
characteristics do not need to be asked within the survey as they will be available from the campus 
mainframe. 
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Attachments 2, 3, and 4 address the creation of evaluation questions and provide examples. 
Attachment 2 outlines suggested dimensions of satisfaction that apply to campus services if the 
college is developing its own survey form. Attachment 3 is an example of a short survey to 
determine level of recognition and satisfaction with campus services. Attachment 4 provides 
examples of questions for a more specific evaluation of matriculation services. Using all of the 
questions included in the attachments would produce a long survey instrument, not unmanageable, 
but perhaps unwelcome in the classroom. Care should be taken when designing your college's 
survey to ensure that the time needed for completion is reasonable. 

Traditional sources of information for students can create "hidden controls" on your study 
population. Proper names of offices instead of informal names can create confusion (e.g., 
"Administration Building," may mean nothing to a student who refers to it as Smith Hall; when in 
doubt, use all known names of offices, including nicknames and abbreviations). If you have 
multiple sites for certain services, provide separate evaluation questions for each site, or ha/e 
students indicate where they receive services (see Attachment 1, question 14). 

If the college has established baseline information about student perceptions and satisfaction with 
support services (from ongoing research, special accreditation surveys, etc.), this project will 
update and broaden such perspectives. Repeat the questions that were asked in previous years in 
order to compare changes in responses over time. 



Subjects 

This study is designed as a classroom survey. Selecting the appropriate sample of your student 
body is essential to the collection of helpful information. Depending on resources, various 
approaches are possible. Knowledge of your curriculum and assistance from Data Processing in 
generating student profiles on different courses will assist in identifying a cross-section of classes 
to allow for a good sampling of the student body and a sufficient sample size on which to base 
your conclusions. All classes on a Tuesday or Wednesday during a specified morning hour and 
one early evening hour could be designated as the classes to be surveyed. Another recommended 
approach is to include primarily introductory classes, which have a dominance of freshman 
students, as the courses in which to administer the survey. 

The "served" matriculants will need to be identified from the non-served students. The targeted 
student group will be the new students, the traditional "freshmen." If your selected subgroup 
population of new freshmen is ur 500, you will want to do a random sample of at least 200 
individuals (see Sampling Guideh, .n Appendix B); if it is smaller, it is desirable to sample all of 
them. Remember, only a select proportion surveyed in the classrooms will be your "targeted" 
population. 

Colleges may wish to identify selected subgroup populations within their student body. 
Comparisons among age, gender, and ethnic groups should be considered. You will need to 
compare the student characteristics of the sample and the college population. Age, gender, 
ethnicity, ar 4 educational goals should be comparable, and the study sample should not differ 
significantly */om the targeted population. 

Procedure 

The survey will be administered in the classroom. It is intended that you survey students late in the 
term (within the week following second census). Care should be taken to involve the instructional 
faculty with the project. The President or other officers of the college should actively support the 
project at meetings and by sending a letter requesting participation. It would also be helpful if the 
survey only took 30 minutes or less to distribute, respond to, and collect so that classroom 
instruction is not disrupted for the entire time. 
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While data processing is likely to be involved in compiling the data generated by the survey, there 
are two options for the mechanical side of the data collection. The first is a survey where students 
can mark their answers with any writing implement on the document; these responses will need to 
be manually entered. The second option is to use a standard optically-read response sheet (which 
can be purchased in bulk) to accompany the survey instrument. While the second option mates 
inputting the data much faster and easier, it does require the use of No. 2 pencils. 



Results 

Organization of the Data 

The initial format for the report should parallel the survey instrument with summary results instead 
of the blanks from which respondents chose (see Attachment 3). A computer will be able to 
summarize the responses. A statistical package will be able to generate more detailed information. 
See Attachment 3 for an example of recognition and satisfaction indices using that approach. 

Examples for organizing the results could include graphs comparing the distributions of response 
by subgroups. 

Analysis of the Data 

Frequency counts and percentages will be calculated for each survey item. 

If respondent identification is planned, a statistical package is beneficial in maximizing research 
possibilities. Use and satisfaction with services could be matched to assessment information and 
academic performance to develop counseling recommendations for students. Additionally, you 
can develop a data base for review after additional semesters of work by the students. 



Discussion 

Suggestions for points to cover in the Discussion section 

1 . Interpretation of the findings. Does the fully matriculated student use campus services more 
frequently, and with greater satisfaction and success than other students? Are there 
differences in use, satisfaction, and success related to age, ethnicity, gender, or educational 
goal? 

2. Limitations of the study. Any survey instrument is only as good as the clarity of its questions, 
sampling representativeness, and student truthfulness. All of these factors should be discussed 
fully in committee. Also, it is recomended that your initial draft of the survey instrument be 
pilot tested on 10-20 students followed by a detailed discussion of the items. This can help 
overcome the usual limitations of most questionnaires. Surveys frequently only allow for 
"bare bones" responses. Have your committee check each item carefully to be sure you are 
getting the information desired. 



Studj Options 

1 . If you have diff' ~ent types of orientation and different sites for counseling, do a comparison of 
the use and satisfaction data to evaluate the effectiveness of the alternative delivery systems. 
The different versions of orientation and counseling will need to be indicated by the student on 
the survey instrument. 



2. If you can match the survey to student records, do regression analyses with grades as the 
outcome variable, and assessment information and service urination or satisfaction as the 
independent variables for different subgroups of the student body. 

Technical Assistance 

Contact person for this design: 

Scot L. Spicer 
Planning and Research 
Glendale Community College 
1500 N. Verdugo Road 
Glendale, California 91208 

Office: (818)240-1000 extension 413 
Home: (213) 839-8380 

Available technical assistance includes consultation on design, execution, data analysis, and 
reporting. 



ATTACHMENT 1 

Demographic Information on Students 



For a classroom survey, a short explanation and request for assistance should precede the questions. For example. The college needs 
information and opinions from students to evaluate and improve programs and services. Your responses arc confidential and will not 
be revealed. We appreciate your cooperation. Please mark one answer for each question or fill in the blank. 

1 . When was your first term at ( ) College? Fall Winter Spring 

Yean 1990 1989 1988 1987 _ 1986 1985 1984 1983 or before 

2. What year did you graduate from high school? 

3 . How old are you ? 

4. How many units will you complete this term? 



5 . How many units have you completed at ( ) College prior to this term? 

6. Gender: Female Male 

7 . What is your residential zip code? 

8 . Which best describes your ethnicity? 

Use the new MIS definitions (see Operational Definitions in Appendix A). You can elaborate to meet your specific needs. 

9 . Were you born in the United S tates? Yes No (An option some colleges may desire.) 

10. Are you a citizen of the United States? Yes No (An option some colleges may desire.) 

1 1. How many hours do ycu work each week? None 1 to 9 

10 to 19 20-29 30-39 40 or more 

12. Which of the following best describes your current educational goal? 

Use the new MIS categories (see Operational definitions in Appendix A). 
(Be sure to use your college's appropropriate terms for the following two questions) 

13. Have you worked with a counselor to develop an educational plan? Yes No 

14. Which one of the following offices do you use most frequendy for assistance in selecting courses and 
planning your education? 

Have never had assistance Financial Aid Office 

Counseling Office EOP/S 



Career Center Other: 



Disabled Student Center (please indicate) 
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ATTACHMENT 2 

Suggested Dimensions of Satisfaction 



Appendices 3 and 4 are examples of "awareness and satisfaction" survey instruments. While these 
formats and forms can be used, staff and faculty involved with the evaluation and study of matriculation 
may well desire to develop their own instrument. In doing so, the following dimensions are suggested for 
gathering helpful evaluation information. 



1. Frequency of use: 

Frequently used services which impact a great number of students may be the most 
important "service centers" to focus on in terms of later strategies for improvement. 

2. Perceived "usefulness" or "satisfaction" with service; a three or five category scale is easy to use: 

— Never visited/used Never visited/used 

— Somewhat useful or - Not at all useful 

— Quite useful Somewhat useful 

Useful 
- Very useful 

3. Awareness/knowledge of services: 

In order to check for one form of response b' s, throw in offices/services that do not exist at your 
college to check reliability of answers (e.g., Student Typing Services). 

4. Assessment of student goals: 

— use MIS categories (See Appendix A) 

5. Areas of usefulness/satisfaction: 

— attitude of staff 

— relation of information to educational goals 

— quality/interest/comprehensibility of written information 

— quality/interest/comprehensibility of verbal information 

— hours of operation 

— availability of staff 

— ease of setting appointments/other measures of access 

— ease of finding office 

— accessibility of office (critical for special needs students and night time students) 

6. Recommendations for improvements: 

— hours of operation 

— shorter lines 

— more staff 

— telephone service 

— more knowledgeable staff 

— availability of inte/preters 
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ATTACHMENT 3 

Campus Services Questionnaire 



This is an example of a satisfaction survey which includes both recognition and satisfaction mdicies. Responses in 
categories B through E "recognize" the service unit, and the ratio of D and E responses to the sum of C, D, and E 
responses (see scale below) gives you a satisfaction index. For example, if the numeric responses for the first item were. 

MLA M.B 3L.C 150 D 100 E 
(20%) (20%) (10%) (30%) (20%) 

Recognit ion is 80% or .80 (B + C + D + E or .20 + .10 + • .20); 

is 60% or .60 (C + D + E or .10 + .30 + .20); and the 
Satisfaction index i s .83 (D + E/C + D + E or .30 + .20/ .10 + .30 + .20). 

Use the names of services/service units as your students % <il identify them. Some of these will not apply to your 
cqUssl 

Please indicate which one of the following statements best describes your experiences with each campus service. 

Use the following scale: (A) Have never heard of it 

(B) Heard of it, but have r^ver used it 

(C) Have used it, but was not satisfied 

(D) Have used it, and found it helpful 

(E) Have used it, and found it very helpful 



Student Computer Center A 

Campus Library A 

Learning Center A 

Writing Lab A 

Math-Science Center A 

Tutorial Center A 

Nursing Learning Center A 

Testing/Assessment Center A 

Job Placement Center A 

Academic Counseling Center A 

EOPS Office A 

Transfer Center A 

Health Center A 

Financial Aid Office A 

Disabled Student Services A 

Admissions and Records A 

Career Center A 



B C D E 

B C D E 

B C D E 

B C . D E 

B C D E 

B C D E 

B C D E 

B C D E 

B C D E 

B C D E 

B C D E 

B C D E 

B C D E 

B C D E 

B C D E 

B C D E 

B C D E 
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ATT ACHMENT 4 

Component Evaluation 



These are questions for a more comprehensive evaluation of matriculation service areas. 
Admissions 

1 . How would you rate the convenience of the Admissions process? 

Excellent Good Fair Poor No Opinion 

2. How would you rate the encouragement given you by the Admissions Office personnel? 

Excellent Good Fair Poor No Opinion 

3. How would you rate the efficiency of registering for classes? 

Excellent GoOv' Fair Poor No Opinion 

4. How long did it take you to register for classes this term? 

15 minutes or less 45 to 60 minutes 

15 to 30 minutes 60 to 90 minutes 

30 to 45 minutes more than 90 minutes 

Orientation 

5. Did you participate in a campus orientation program or orientation class? 

Yes No (If No, go to question #10) 

6. How long was your orientation session? OR (Optional - Individual colleges should choose appropriate responses) 
Which of the following types of orientation did you go through? 

7. How would you rate the helpfulness of the information given at your Orientation session? 

Excellent Good Fair Poor No Opinion 

8. Did attending the Orientation session influence you to visit other campus services? 

Yes No 

9. Did attending the Orientation session help you in choosing your classes? 

Yes No 

Assessment 

10. Have you undergone any testing at this college to determine your reading, writing and math skills? 

Yes No (If No, go to question #14) 

1 1. Do you believe your course placement in English was appropriate? 

Yes No Undecided/Not applicable 

12. Do you believe your course placement in math was appropriate? 

Yes No Undecided/Not applicable 

1 3. Did a member of the college staff explain your results to you? 

Yes No 

14. Have you completed any tests at the college to determine your vocational/career interests? 

Yes No (If No, go to question #16) 

15. How would you rate the helpfulness of the testing you have done in selecting courses or programs at the collets? 

Excellent Good Fair Poor No Opinion 
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Counseling/Advisement 

16. Have you met will) a counselor? 

Yes No (If No, go to question #22) 

17. Did the counselor assist you in choosing courses and programs? 

Yes No 

18. How would you rate the he ? />fulncss of the counselor you have seen? 

Excellent Good Fair Poor No Opinion 

19. Did your counseling appointment help clarify your educational goals and how to achieve them? 

Yes No 

20. Did your counselor help you understand about course and basic skills prerequisites? 

Yes No 

21. Did your counselor refer you to other campus services? 

Yes No 

Overall/Goals 

22. When did you first decide upon a specific educational goal to pursue at this college? 

Before applying to the college During my second, third term, etc. 

During my orientation session I'm still undecided 

During my first term at the college 

23. Have you chosen or changed your educational goal? 

Yes No (If No, go to question #25) 

24. What factors led you to identify or change your goal? (Check all that apply) 

The results of the English and math assessments 

The results of vocational tests I took 

Discussions with faculty in my courses 

Discussions with counselors 

Discussions with other staff 

Did not do as well as expected in the major originally selected 

Found I did not enjoy the courses in the major originally selected 

Discovered an area of study that I enjoyed more 

Lack of job opportunities in the area originally selected 

Job/Career opportunities in another area were of greater interest to me 

The college aid not offer a program/course in my area of interest 

Family responsibilities 

Job responsibilities 

Other: 



(please state reason) 



Follow-up 
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25. What will your student status be next term? 

Continuing at this college 

Continuing at a 4-ycar college 

Continuing at another community college 

Wi M . have completed my educational goal 

Temporary leave, but will return 

Unsure at this time 
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APPENDIX A 



Matriculation Local Research Options Project 
Operational Definitions 



Term 



Definition 



Successful 
Nonsuccessful 

Success Rate 

Persistence 
Peisistence Rate 
Retention 
Retention Rate 



Matriculated 
<r 



Exempt Student 
Drop-out 

Remedial Reading 
Remedial Writing 
Remedial Math 
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Grades of A, B, C, and CR represent success. 

D, F, NC, W, and I grades represent nonsuccess. Grades of In Progress (DP), 
Report Delayed (RD), and other notations of no grade of record should be excluded 
(e.g., Drop-DR). 

Number of successful units divided by the total number of successful plus 
nonsuccessful units. Multiply this proportion by 100 to convert the number to a 
percentage. 

A student persists to the subsequent term when any grade (excluding no shows) 
appears on the transcript for the subsequent term. 

Out of an original specified fc .oup, it is the percentage of students who persisted to 
the subsequent term. 

A student is retained in a class when the transcript shows a grade of A, B, C, CR, 
D, or I on the student's transcript at the end of the term. 

The total number of .nrolled units with grade notations of A, B, C, CR, D, or I 
divided by the total number of enrolled units with grade notations of A, B, C, CR, 
D, F, NC, W, and I, excluding grades of RD, IP and other notations indicating no 
grade of record (e.g., DR) Multiply this proportion by 100 to convert the number 
to a percentage. 

For purp oses of this project, a matriculated student is one who has received the 
-^Student/following services: admissions, assessment, orientation, and counseling/ 
advisement. These services must have been rendered prior to the second week of 
the term. Students enrolled in an on-going matriculation class would be considered 
a matriculated student. This definition should not be construed to be the official 
definition of a "matriculated" student. 

An exempt student is one who is admitted but not required to participate in the 
assessment, orientation, or counseling/advisement process as specified in the local 
district policy of exemption. 

A student who, within a given term, has officially dropped all units prior to the end 
of the term. 

Non AA-applicable reading courses designed to prepare students for college-level 
reading. 

English and ESL writing courses two or more levels below Freshman Composition 
(English 1A). F 

Math Com; °, in any department below Beginning or Introductory Algebra. 
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Ethnic 
Status 



Asian 

Black Non-Hispanic 

Filipino 

Hispanic 

American Indian/Alaskan Native 



Other Non-white 
Pacific Islander 
White Non-Hispanic 
Unknown/Non-Rcspondent 



These categories are from the June '89 MIS Data Element Dictionary (SB05). 
Colleges can use the expanded MIS version for their local study although we would 
like the data to be collapsed for state-wide reporting. Colleges also may add 
categories of ethnic groups for their own local use. 

Use these categories for ethnic breakdowns if possible, otherwise, use the ones 
currently in use in your district. 



Age 



Less than 20 years old 
20 - 24 years old 
25 - 29 years old 
30 - 49 years old 
50 or more years old 



Credit 
Load 



0.1 -5.9 units 
6.0- 11.9 units 
12.0 or more units 



Educational Goals 



The educational goals (SB14 and SM01 from the June '89 MIS Dictionary) to be 
used in the project are as follows: 

Obtain a Bachelor's Degree after completion of an Associate's Degree - BA & AA 
Obtain a Bachelor's Degree without completing an Associate's Degree - BA w/o AA 
Obtain a two-year Associate's Degree without transfer - AA w/o Transfer 
Obtain a two-year vocational degree without transfer - Voc. AA w/o Transfer 
Obtain a vocational Certificate without transfer - Cert, w/o Transfer 
Discover/formulate career interests, plans, goals - Discover Interests 
Prepare for a new career (acquire job skills) - Prepare Career 
Advance in current job/career (update job skills) - Advance Career 
Maintain certificate or license (e.g., Nursing, Real Estate) - Maintain Cert./Lic. 
Educational development (intellectual, cultural) - Educ. Dev. 
Improve basic skills in English, reading or math - Improve Bas. Sk. 
Complete credits for high school diploma or GED - Complete H.S. units 
Undecided on goal - Undecided 



Other 

Demographic 
Indicators: 
Work hours, 
Gender, etc. 



Use the definitions and breakdowns found in the June 1 89 MIS Data Element 
Die/ mary so we can have uniformity in reporting. 



ERLC 
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APPENDIX B 



Sampling Guidelines 



The primary concern in any sampling procedure is to be confident that the sample selected is representative 
of the population from which the sample is dra" \. "Sampling error" is the degree to which the data from 
a random sample deviates from data that woulo oe obtained from the whole population. 

The first step in sample size selection is to determine the "cc. fidence level" desired. The confidence level 
typically used in social science research is .05. This means that there is only a 5% probability that 
observed group differences are solely due to chance factors or sampling error. 

Generally, larger samples tend to be more representative of the total population. Nevertheless, overly 
large samples are unnecessary because it will not significantly decrease the sampling error (sampling error 
is inversely related to the square of the sample size). 

For experiments involving random assignment of subjects to treatments, a sample size of at least 100 per 
treatment group will reveal any treatment effects that are iarge enough to matter educationally. For surveys 
or other studies dealing with population proportions, Table 1 presents the sample sizes required in order 
for sample proportions to be within .05 of the true population proportions. 

When sampling is used, it is advisable to gather certain demographic information about the sample of 
students (e.g., ethnicity, age, sex, educational goal) to ensure that the sample is indeed representative of 
the college population on these characteristics. It is suggested that frequency distributions of these 
demographic data be presented for the sample (and if possible, for the college) in the Results section of 
each study that uses sampling. 

When observations are made about subgroups of the total sample, very small subgroups are likely to 
misrepresent their subpopulation. It may be necessary to increase overall sample size to insure that each 
subgroup contains at least 30-50 students. Alternatively, stratified sampling approaches may be used. 
Stratified sampling entails random sampling within a subgroup population to ensi-re that you have the 
appropriate proportion of the subgroup represented within the total sample. 



Table 2 is a list of random numbers to help select your sample. An alternative method that is 
approximately random is to select every Nth name (until the necessary sample size has been reached) 
where N equals population size divided by sample size (e.g., population = 2000; sample = 322; select 
every 6th name from the defined population). 

When sampling techniques are used with survey research, you will need to take into account the expected 
response rate. Generally, classroom surveys have a 70% response rate; therefore, the size of the sample 
will need to be increased by 30%. 
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TABLE 1 

Choosing a Sample Size at a 95% Confidence Level 



Statistical Techniques and the Analysis of Data/ / 93 



TABLL FOR DETERMINING NEEDED SIZES OF A RANDOMLY CHOSEN SAMPLE 
FROM A GIVEN FINITE POPULATION OF N CASES SUCH THAT THE SAMPLE 
PROPORTION p WILL BE WITHIN ± .05 OF THE POPULATION PROPORTION 
P WITH A 95 PERCENT LEVEL OF CONFIDENCE 1 



A/ 


c 
o 


a; 

/V 


S 


N 


S 


10 


10 


220 


140 


1200 


29 i 


15 


14 


230 


144 


1300 


297 


:o 


19 


240 


148 


1400 


302 


25 


24 


250 


152 


1500 


306 


30 


28 


260 


155 


1600 


310 


35 


32 


270 


159 


1700 


313 


AO 


36 


280 


162 


1800 


317 


AS 


40 


290 


165 


1900 


320 


50 


44 


300 


169 


2000 


322 


55 


48 


320 


175 


2200 


327 


00 


52 


3-10 


181 


2400 


331 


65 


56 


360 


186 


2600 


335 


70 


59 


380 


191 


2800 


338 


75 


63 


400 


196 


3000 


341 


80 


66 


4?,0 


201 


3500 


346 


85 


70 


440 


205 


4000 


351 


90 


73 


460 


210 


4500 


354 


95 


76 


480 


214 


5000 


357 


100 


80 


500 


217 


6000 


361 


110 


86 


550 


226 


7000 


364 


120 


92 


600 


234 


8000 


367 


130 


97 


650 


242 


9000 


368 


M0 


103 


700 


248 


10000 


370 


150 


108 


750 


254 


15000 


375 


160 


113 


800 


260 


20000 


377 


170 


118 


850 


265 


30000 


379 


180 


123 


900 


269 


40000 


380 


190 


127 


950 


274 


50000 


381 


200 


132 


!000 


278 


75000 


382 


210 


136 


1100 


285 


100000 


384 



Note: N is population size; S is sample sue. 



I. Krejcie, R. V. ami Morgan, D. W. Determining sample size Tor research activities, Educational and 
Psychological Mensuremnu, 1 970, 30, 607-6 1 0. 
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TABLE 2 

Random Sampling Techniques 



Obtain a list of all members of your population. Assign each member of the population a number (e.g., 
l,2,3,etc). Determine your desired sample size (N). Now enter the table of random numbers and 
arbitrarily select a column and row as your starting number. Combine rows if vnu need 3 or more digits. 
For example, suppose you want to select 200 students from a population of 10,000 and you pick 10 09 as 
a starL.ig point. The next student selected should be 7325, then 3376, etc. 



10 


00 


73 


25 


33 


76 


52 


01 


35 


86 


34 


67 


35 


.8 


76 


80 


95 


90 


91 


17 


39 


29 


27 


49 


45 


3? 


54 


50 


48 


05 


64 


89 


47 


42 


96 


24 


80 


52 


40 


37 


20 


63 


61 


04 


02 


00 


82 


29 


16 


66 


08 


42 


26 


89 


53 


19 


64 


50 


93 


03 


23 


20 


90 


25 


60 


15 


95 


33 


47 


64 


35 


08 


03 


35 


06 


W 


01 


90 


25 


29 


09 


37 


67 


07 


15 


38 


31 


13 


11 


65 


88 


67 


67 


43 


97 


04 


43 


62 


76 


59 


12 


80 


79 


99 


70 


80 


15 


73 


61 


47 


64 


03 


23 


66 


53 


?8 


95 


11 


68 


77 


12 


17 


17 


68 


33 


66 


06 


57 


47 


17 


34 


07 


27 


68 


50 


36 


69 


73 


61 


70 


65 


81 


33 


98 


85 


11 


19 


92 


91 


70 


31 


06 


01 


08 


05 


45 


57 


18 


24 


06 


35 


30 


34 


26 


H 


86 


79 


90 


74 


39 


23 


O 


30 


97 


32 


65 


26 


97 


76 


02 


02 


05 


16 


66 


92 


68 


66 


57 


48 


18 


73 


05 


38 


52 


47 


18 


62 


38 


85 


79 


63 


57 


33 


21 


35 


05 


32 


54 


70 


48 


90 


55 


35 


75 


48 


28 


46 


82 


87 


09 


83 


49 


12 


56 


24 


73 


79 


64 


57 


53 


03 


52 


96 


47 


78 


35 


80 


83 


42 


82 


60 


93 


52 


03 


44 


35 


27 


38 


84 


35 


98 


52 


01 


77 


67 


U 


90 


56 


86 


07 


22 


10 


94 


05 


58 


60 


97 


09 


34 


33 


50 


50 


07 


39 


98 


11 


80 


50 


54 


31 


33 


80 


82 


77 


32 


50 


72 


56 


82 


48 


29 


40 


52 


42 


01 


52 


77 


56 


78 


51 


S3 


45 


29 


96 


34 


06 


28 


89 


80 


83 


13 


74 


67 


00 


78 


18 


47 


54 


06 


10 


68 


71 


17 


78 


17 


88 


6$ 


64 


02 


00 


86 


50 


75 


84 


01 


36 


76 


66 


79 


51 


90 


36 


47 


64 


93 


29 


60 


91 


10 


62 


99 


59 


46 


73 


48 


C7 


51 


76 


49 


69 


91 


82 


60 


89 


28 


93 


78 


56 


13 


68 


23 


47 


83 


41 


13 



65 


43 


12 


76 


74 


80 


12 


43 


56 


35 


74 


35 


09 


98 


17 


69 


91 


62 


68 


03 


09 


80 


32 


05 


05 



17 


46 


85 


09 


50 


17 


72 


70 


80 


15 


77 


40 


27 


72 


14 


66 


25 


22 


91 


48 


14 


22 


56 


85 


14 



58 


04 


77 


69 


74 


45 


31 


82 


23 


74 


43 


23 


60 


02 


10 


36 


93 


68 


72 


03 


46 


42 


75 


67 


88 



73 


03 


95 


71 


86 


21 


11 


57 


82 


53 


45 


52 


16 


42 


37 


76 


62 


11 


39 


90 


96 


29 


77 


88 


22 



40 


21 


81 


65 


44 


14 


38 


55 


37 


63 


96 


28 


60 


26 


55 


94 


40 


05 


64 


18 


54 


38 


21 


45 


98 



91 


49 


91 


45 


23 


80 


33 


69 


45 


98 


44 


10 


48 


19 


49 


12 


55 


07 


37 


42 


63 


60 


64 


93 


29 



68 


47 


92 


76 


36 


26 


94 


03 


£8 


58 


85 


15 


74 


7* 


54 


11 


10 


00 


20 


40 


16 


50 


53 


44 


84 



46 


16 


28 


35 


54 


70 


29 


73 


41 


35 


32 


97 


92 


65 


75 


12 


86 


07 


46 


97 


40 


21 


95 


25 


63 



9v 


,5 


08 


99 


23 


53 


14 


03 


33 


40 


57 


60 


04 


08 


81 


96 


64 


48 


94 


39 


43 


65 


17 


70 


82 



37 


08 


92 


00 


48 


42 


05 


08 


23 


41 




22 


20 


64 


13 


28 


70 


72 


58 


15 


07 


20 


73 


17 


90 



61 


19 


69 


04 


46 


15 


47 


44 


52 


66 


94 


55 


72 


85 


73 


42 


48 


11 


62 


13 


23 


52 


37 


83 


17 



20 


45 


74 


77 


74 


95 


27 


07 


99 


53 


67 


89 


75 


43 


87 


97 


34 


40 


87 


21 


73 


20 


88 


98 


37 



51 


92 


43 


37 


29 


59 


36 


78 


38 


48 


54 


62 


24 


44 


31 


18 


86 


84 


87 


67 


68 


93 


59 


14 


16 



65 


39 


45 


95 


93 


82 


39 


61 


01 


18 


91 


19 


04 


25 


92 


03 


07 


U 


20 


59 


26 


25 


22 


96 


63 



42 


58 


26 


05 


27 


33 


21 


15 


94 


66 


92 


92 


74 


59 


73 


25 


70 


14 


66 


70 


05 


52 


28 


25 


62 



MC 



04 


49 


35 


24 


94 


75 


24 


63 


38 


24 


45 


86 


25 


10 


25 


61 


96 


27 


93 


35 


65 


33 


71 


24 


72 


00 


54 


99 


76 


54 


64 


05 


18 


81 


59 


96 


11 


96 


38 


96 


54 


69 


28 


23 


91 


23 


28 


72 


95 


29 . 


35 


96 


31 


53 


07 


26 


89 


80 


93 


54 


33 


35 


13 


54 


62 


77 


97 


45 


00 


24 


90 


10 


33 


93 


33 ; 


59 


80 


80 


83 


9! 


45 


42 


72 


68 


42 


83 


60 


94 


97 


00 


13 


02 


12 


48 


92 


78 


56 


52 


01 


06 j 


46 


05 


88 


52 


36 


01 


39 


00 


22 


86 


77 


28 


14 


40 


77 


93 


91 


08 


36 


47 


70 


61 


74 


29 


41 ; 


32 


17 


90 


05 


97 


87 


37 


92 


52 


41 


05 


56 


70 


70 


07 


86 


74 


31 


71 


57 


85 


39 


41 


18 


38 \ 


69 


23 


46 


14 


06 


20 


11 


74 


52 


04 


15 


95 


66 


00 


00 


18 


74 


39 


24 


23 


97 


11 


89 


63 


38 j 


19 


56 


54 


14 


30 


01 


75 


87 


53 


79 


40 


42 


92 


15 


85 


66 


67 


43 


68 


06 


84 


96 


28 


52 


07 ; 


45 


15 


51 


49 


38 


19 


47 


60 


72 


46 


43 


66 


79 


45 


43 


59 


04 


79 


00 


33 


20 


82 


66 


95 


41 j 


94 


86 


43 


19 


94 


36 


16 


81 


08 


51 


34 


88 


88 


15 


53 


01 


54 


03 


54 


56 


05 


01 


'.5 


11 


76 j 



Source. From tables of the Rand Corporation from A Million Random Digits with 100,000 Nonnol Deviates 
(New Yon<: The Free Press, 1955) by permission of the Rand Corporation. 
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